Method for deriving at least one item of information from images of a stereo camera

ABSTRACT

A method for deriving at least one item of information from images of a stereo camera. A resource-saving and at the same time accurate derivation of information from the images is achieved by alternately using and processing the images with methods of monocular image processing and deriving at least one item of information from the results. A motor vehicle comprising a stereo camera and comprising a control device which carries out the method, are also described.

CROSS REFERENCE

The present application claims the benefit under 35 U.S.C. § 119 ofGerman Patent Application No. DE 10 2022 200 542.2 filed on Jan. 18,2022, which is expressly incorporated herein by reference in itsentirety.

FIELD

The present invention relates to a method for deriving at least one itemof information from images of a stereo camera. The invention alsorelates to a motor vehicle comprising a stereo camera and a controldevice for carrying out the method.

BACKGROUND INFORMATION

In many applications, it is desirable to obtain information about thecurrent environment of the application. One such application is a motorvehicle, for example. For this purpose, at least one camera whichcaptures images of the environment of the application during operationcan be used. Information is then derived from the images. Depthestimation in particular is of special interest here, because it isimportant for driving assistance, for example, and/or for at leastpartially autonomous driving of motor vehicles.

The use of a single camera for this purpose is available in the relatedat. In monocular systems, information is derived from the images of thecamera by processing the successive images of the camera using methodsfor monocular image processing.

The use of a stereo camera comprising two cameras is available for thispurpose as well, wherein the respective camera captures associatedimages. In the corresponding stereo methods, the new images of bothcameras are used and processed at each point in time. In particular thestereo disparity between simultaneously captured images and/or anoptical flow between successively captured images is determined. Thisallows improved depth estimation and improved detection of self-movingobjects.

Corresponding systems and methods are described, for example In GermanPatent Application No. DE 10 2012 014 994 A1 and European PatentApplication No. EP 1 580 092 A2.

The monocular systems and methods in particular have the disadvantagethat a depth estimation for self-moving objects is not possible or issubject to high error rates. The stereo systems and methods inparticular have the disadvantage that the processing of the imagesinvolves a high consumption of resources.

SUMMARY

The present invention has an object of providing improved or at leastdifferent embodiments for a method for deriving at least one item ofinformation from images of a stereo camera and for a motor vehiclecomprising such a stereo camera, which in particular may eliminatedisadvantages from the related art.

This object may be achieved according to features of the presentinvention. Advantageous embodiments of the present invention aredisclosed herein.

The present invention includes alternately using and processing theimages captured by cameras of a stereo camera at successive points intime in order to derive information from the images. Therefore, at eachsuccessive point in time a new image has to be taken into account. Thismeans that methods designed for monocular processing are applied to analternating image sequence of a stereo camera. Compared to the stereomethods available in the related art, this results in a considerablereduction of required resources. This in particular saves bandwidthsand/or memory and/or the consumption of electrical energy. This inparticular avoids the calculations of disparities between simultaneouslycaptured stereo images and/or calculations of the change over timebetween pairs of successive images which are usually necessary in stereomethods. Compared to stereo methods, the solution according to theinvention further offers the advantage that two different methods, inparticular algorithms, are not required. Compared to monocular systemsavailable in the related art, there can be no linear movement of thecameras at successive points in time. As a result, improved informationcan be derived from the environment, in particular from objects movingin parallel. Compared to monocular systems, there is furthermore theadvantage that a pseudo movement can be determined even when stationary,which can be advantageous for a depth estimation. Another advantage overmonocular systems is that, due to the usually known distance between thetwo cameras of the stereo camera, there is a fixed reference value that,for example, makes it possible to eliminate or at least reduce theproblem of scale ambiguity of monocular systems.

In accordance with an example embodiment of the present invention, in astep which hereinafter is also referred to as a monocular step, theimages of the first camera and the second camera are used and processedwith methods of monocular image processing alternately in terms of timeand output as results in order to derive at least one item ofinformation from images of a stereo camera comprising a first camera anda second camera. In a step, which hereinafter is also referred to as aderivation step, at least one item of information is derived from theresults of the monocular step.

During operation, the first camera captures first images and the secondcamera captures second images. In the monocular step, therefore, thefirst images and the second images are alternately used and processedwith methods of monocular image processing.

The stereo camera advantageously has properties typical of stereocameras. The cameras are in particular similarly aligned and haveoverlapping fields of view, in particular similar fields of view. Thecameras in particular have an intrinsic alignment. Thus, compared to astereo method, a high-precision extrinsic alignment and calibration ofthe cameras can be omitted or at least carried out to a lesser extent;in particular since the cameras comprise an intrinsic calibration and/orrectification. This requirement can essentially also be achieved withimage preprocessing.

It is preferable if the distance between the cameras is known and istaken into account in the method. This leads to improved accuracy inboth the monocular step and the derivation step.

The method can be simplified if both cameras are alike, in particularidentical.

“Methods of monocular image processing” is in particular intended to beunderstood here as processing the images in such a way that, compared toa stereo method, the successive images of the two cameras are not usedsimultaneously but rather that the images of the first camera and thesecond camera are used alternately. Consequently, only one new image hasto be processed at each point in time.

The monocular step can include any methods of monocular imageprocessing.

Advantageously, in the monocular step, an optical flow of thealternately used first images and second images is determined and outputas a result.

According to an example embodiment of the present invention, it ispreferable if, in the monocular step, at least one image feature of thealternately used first images and second images is tracked over time andoutput as a result.

Alternatively or additionally, monocular deep learning methods can beused in the monocular step.

The derivation step can be used to derive any information from theresults of the monocular step.

Preferably, information for depth estimation is derived in thederivation step.

In advantageous embodiments of the present invention, at least onemonocular method for deriving at least one of the at least one item ofinformation is used in the derivation step. The resources required inthe derivation step are thus reduced as well.

According to an example embodiment of the present invention, preferably,a movement of the stereo camera itself is derived as information in thederivation step.

According to an example embodiment of the present invention, preferably,an adjustment is made when the self-movement is derived. The, preferablyknown, offset between the cameras can be taken into account. If theresult of a previously determined self-movement is assumed for theself-movement, the offset has to be taken into account twice, i.e. addedor subtracted, in particular because the direction has to be reversed.

Alternatively or additionally, the derivation step can include warping.Advantageously, images and/or depth measurement values and/orpredications and/or features and/or certainties are warped, preferablyin accordance with a depth map and/or another type of predication whichcan be both forward and backward in time.

In advantageous embodiments of the present invention, the estimation ofdepths and/or the detection of objects, in particular self-movingobjects in the environment of the stereo camera are derived asinformation in the derivation step. Changes in the image can beestimated as well.

According to an example embodiment of the present invention, a computerprogram product, for example an algorithm and/or a software, which isappropriately configured, is advantageously used to carry out the methodaccording to the present invention.

The method according to the present invention can be carried out bymeans of a computer system. The computer system is in particularconfigured in such a way that, during operation, it executes thecomputer program product in order to carry out the method.

In principle, the method can be used in any applications.

The method is advantageously used in a motor vehicle. The motor vehiclethus comprises the stereo camera. The motor vehicle further comprises acontrol device for carrying out the method, which is configuredaccordingly. In other words, the control device is configured in such away that it derives at least one item of information from the images ofthe stereo camera according to the method.

The control device can comprise the computer system and/or be acomponent of the computer system. The control device can in particularat least partially include the computer program product.

The at least one derived item of information is advantageously madeavailable to a driving assistance system of the motor vehicle. Thedriving assistance system is configured to assist an operator and/or forat least partially autonomous driving of the motor vehicle, wherein thedriving assistance system takes the at least one item of informationinto account for this purpose.

According to an example embodiment of the present invention, the stereocamera is advantageously disposed in the motor vehicle in such a waythat it captures images in the front area of the motor vehicle when themotor vehicle is traveling forward. The reliability of the drivingassistance system can thus be increased. The stereo camera can inparticular be mounted on a windshield of the motor vehicle.

Compared to a monocular system, the method according to the presentinvention may further offer the advantage that the two cameras of thestereo camera can also be used for initialization and/orreinitialization with a known self-movement. A reinitialization can inparticular be carried out when the method and/or the stereo cameraand/or the motor vehicle is started, or after an error has been detectedor in the event of an implausible result. For this purpose, in deviationfrom the above-described method for deriving the information,simultaneously captured images can be taken into account. It is possibleto use stereo methods for initialization or reinitialization. It is inparticular possible to circumvent or suspend the derivation of theself-movement and assume a respective ground truth if the relativeposition between the cameras is known with sufficient accuracy.Reinitialization can be used, in particular periodically, forvalidation.

Sources of said errors which can make initialization or reinitializationnecessary, are, for example, high speeds at which very much changes inthe successive images and/or the brief failure of at least one of thecameras and/or a brief occlusion of at least one of the cameras, forexample due to rain or debris, or a decalibration of at least one of thecameras.

Further main features and advantages of the present invention willemerge from the disclosure herein, including from the figures and fromthe associated description of the figures.

It goes without saying that the aforementioned features and the featuresyet to be explained in the following can be used not only in therespectively specified combination, but also in other combinations or ontheir own, without leaving the scope of the present invention.

Preferred embodiment examples of the present invention are shown in thefigures and will be explained in more detail in the followingdescription, wherein the same reference numbers refer to the same orsimilar or functionally the same components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a greatly simplified circuit diagram-like illustration of astereo camera in a motor vehicle,

FIG. 2 shows a symbolic illustration of images captured with the stereocamera,

FIG. 3 shows a flow chart to explain a method for deriving at least oneitem of information from images of the stereo camera, according to anexample embodiment of the present invention.

FIG. 4 shows images of the camera.

FIG. 5 shows a difference between images of FIG. 4 according to therelated art.

FIG. 6 shows a difference between images of FIG. 4 according to thepresent invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

A stereo camera 3 comprising a first camera 1 and a second camera 2, asshown greatly simplified in FIG. 1 , is in particular used in a motorvehicle 100, as shown as an example and greatly simplified and in themanner of a circuit diagram in FIG. 1 . During operation, the firstcamera 1 captures first images 4 and the second camera 2 captures secondimages 5. The images 4, 5 are shown in FIG. 2 , wherein the first images4 are shown symbolically as rectangles and the second images 5 astriangles. In FIG. 4 , examples of images 4, 5 are shown greatlysimplified.

As is typical for stereo cameras, the two cameras 1, 2 have an intrinsicalignment to one another, wherein the distance between the cameras 1, 2is preferably known and is taken into account in the method described inthe following. The cameras 1, 2 furthermore have an at least similaralignment and similar overlapping fields of view. The cameras 1, 2 arepreferably also configured in the same way, in particular identically.

As indicated in FIG. 2 and shown as an example by the flow chart in FIG.3 , at least one item of information, in particular about theenvironment of the stereo camera 3 and thus the motor vehicle 100, isderived from the images 4, 5. For this purpose, the first images 4 andthe second images 5 are used alternately and processed with methods ofmonocular image processing. A time axis is shown as an arrow in FIG. 2 .In the shown example, therefore, both cameras 1, 2 respectively capturesuccessive images 4, 5. In other words, at successive points in time tN,wherein N is a natural number greater than or equal to 0, the firstcamera 1 captures a first image 4 and the second camera 2 captures asecond image 5. As indicated in FIG. 2 with dashed lines, not bothcaptured images 4, 5 are used at the points in time tN; rather a firstimage 4 of the first camera 1 and a second image 5 of the second camera2 are alternately used and processed with methods of monocular imageprocessing.

According to the flow chart shown in FIG. 3 , this is carried out in astep 20, which hereinafter is also referred to as a monocular step 20.In the monocular step 20, therefore, the first images 4 and the secondimages 5 are alternately used and processed with methods of monocularimage processing. A new image 4, 5 is thus processed at each point intime tN. Compared to stereo methods available in the related art (notshown), in which both new images 4, 5 have to be processed at each pointin time tN, this results in a significant reduction in the requiredresources.

The result of this image processing is transmitted to a subsequent step21. In the step 21, at least one item of information is derived fromresults of the monocular step 20. The step 21 is also referred tohereinafter as the derivation step 21.

As can further be seen in FIG. 3 , the monocular step 20 in the shownembodiment example includes a step 22, in which the images 4, 5 are usedalternately as described, and a step 23, in which the respectivecurrently being used image 4, 5 is processed with methods of monocularimage processing. The step 22 is hereinafter also referred to asalternating step 22 and the step 23 is referred to as the processingstep 23. As can further be seen in FIG. 3 , after the derivation step21, the method can return to the monocular step 20, in particular thealternating step 22.

The advantages of the method according to the invention over a monocularsystem consisting of only one camera are explained in the following withreference to FIGS. 4 to 6 as examples.

FIG. 4 shows the time axis with an arrow, wherein images 4, 5 are shownat two successive points in time t0 and t1. Images 4 of the first camera1, which can be a left camera 1, for example, are shown at both pointsin time t0, t1, and an image 5 of the second camera 2, which can be aright camera 2, for example, is shown at the point in time t1. If onlyone camera 1, 2 were provided, the system would be a monocular system.In the following, it is assumed purely as an example that the monocularsystem consists of the first camera 1. This results in the differenceindicated in FIG. 5 between the first images 4 of the monocular systemconsisting of the first camera 1 which are captured at the point in timet0 and t1. The method according to the invention, on the other hand, isused to create the difference between the first image 4 captured by thefirst camera 1 at the point in time t0 and the second images 5 capturedby the second camera 2 at time t1, as indicated as an example in FIG. 6. As a comparison between FIGS. 5 and 6 shows, the difference in FIG. 6and thus in the method according to the invention is significantly morepronounced than the difference in FIG. 5 and thus according to therelated art. The method according to the invention therefore makes itpossible to derive information from the images 4, 5 more reliably, moreaccurately and in a significantly simplified manner. It is in principlealso possible to capture only one image 4, 5 with the camera 1, 2relevant to the time tN at any time tN.

The monocular step 20, in particular the processing step 23, preferablyincludes determining an optical flow from the alternately used firstimages 4 and second images 5, i.e. similar to the difference shown inFIG. 6 . Alternatively or additionally, monocular deep learning methodsare used in the monocular step 20, in particular in the processing step21.

Advantageously, at least one monocular method for deriving at least oneof the at least one item of information is used in the derivation step21.

In the derivation step 21, for example, a movement of the stereo camera3 itself and thus of the motor vehicle 100 is derived as information.Alternatively or additionally, the derivation step 21 can includewarping. This involves, for example, warping the images 4, 5, depthmeasurement values, predications, features or certainties, preferably inaccordance with a depth map and another type of predication which can beboth forward and backward in time. The derivation step 21 alternativelyor additionally includes the estimation of depths and/or the detectionof objects 10 (see FIG. 4 ), in particular self-moving objects 11, inthe environment of the stereo camera 3 and thus the motor vehicle 100 asderived information. The derivation step 21 can also include estimatinga change in the images 4, 5 as information.

As indicated in FIGS. 1 and 3 , the at least one derived item ofinformation can be provided to a driving assistance system 102 of themotor vehicle 100. The driving assistance system 102 is configured toprovide assistance when driving the motor vehicle 100 and/or for atleast partially autonomous driving of the motor vehicle 100, wherein thedriving assistance system 102 takes the at least one derived item ofinformation into account for this purpose.

As can be seen in FIG. 1 , for carrying out the method, the motorvehicle 100 comprises a control device 101 which is connected to thestereo camera 3 and is configured accordingly. The control device 101can comprise a not depicted computer system, for example, or it can be acomponent of the computer system. The control device 101 or the computersystem can include an appropriately configured computer program productfor carrying out the method.

In the embodiment example shown in FIG. 1 , the stereo camera 3 ismounted in the front in a Z-direction 103 of the motor vehicle 100, forexample on a not depicted windshield. Thus, during operation, thecameras 1, 2 capture images 4, 5 from the front area, in particular fromthe front area in the forward direction of travel, of the motor vehicle100. In a driving situation shown in FIG. 4 , objects 10, such asself-moving objects 11, can be captured in the images 4, 5 as explainedabove and their depths, in particular their relative position to thestereo camera 3 and consequently to the motor vehicle 100, can bederived.

What is claimed is:
 1. A method for deriving at least one item ofinformation from images of a stereo camera including a first camera anda second camera, the method comprising the following steps: capturing,by the first camera, first images, and capturing, by the second camera,second images; in a monocular step, alternately using and processing thefirst images and the second images with methods of monocular imageprocessing, and outputting results of the monocular step; and in aderivation step, deriving at least one item of information from resultsof the monocular step.
 2. The method according to claim 1, wherein inthe monocular step, an optical flow of the alternately used first imagesand second images is determined and output as a result.
 3. The methodaccording to claim 1, wherein, in the monocular step, at least one imagefeature of the alternately used first images and second images istracked over time and output as a result.
 4. The method according toclaim 1, wherein monocular deep learning methods are used in themonocular step.
 5. The method according to claim 1, wherein at least onemonocular method for deriving at least one of the at least one item ofinformation is used in the derivation step.
 6. The method according toclaim 1, wherein, in the derivation step, a movement of the stereocamera itself is derived as information.
 7. The method according toclaim 1, wherein the derivation step includes warping.
 8. The methodaccording to claim 1, wherein the derivation step derives an estimationof depths and/or detection of self-moving objects, in an environment ofthe stereo camera as information.
 9. A non-transitory computer-readablestorage medium on which is stored a computer program for deriving atleast one item of information from images of a stereo camera including afirst camera and a second camera, the computer program, when executed bya computer, causing the computer to perform the following steps:capturing, by the first camera, first images, and capturing, by thesecond camera, second images; in a monocular step, alternately andprocessing the first images and the second images with methods ofmonocular image processing, and outputting results of the monocularstep; and in a derivation step, deriving at least one item ofinformation from results of the monocular step.
 10. A motor vehicle,comprising: a stereo camera including a first camera which capturesfirst images during operation, and a second camera which captures secondimages during operation; and control device configured to derive atleast one item of information from the first and second images of thestereo camera, the control device configured to: in a monocular step,alternately use and process the first images and the second images withmethods of monocular image processing, and outputting results of themonocular step; and in a derivation step, derive at least one item ofinformation from results of the monocular step.